The present invention relates to method and apparatus for capturing an image of a scene, and, more particularly, to method and apparatus for capturing a relatively high dynamic range image using a relatively low dynamic range image sensor.
Virtually any real world scene produces a very large range of brightness values. By contrast, known image sensing devices have very limited dynamic ranges. For example, it is typical for a video image sensor to provide only 8-bits or less of grey-scale or color information. In the case of grey-scale images, 8-bits allows only 256 discrete grey-scale levels, which is not sufficient to capture the fine details of most real life scenes.
A known solution to the problem of capturing high dynamic range images with a low dynamic range image sensor is to capture multiple images of each local area of the scene while varying the exposure to light from the scene. There are three general variations of this solution: (1) temporal variation of the exposure while capturing a sequence of images, (2) using multiple image sensors having different sensitivities to light from the scene and (3) spatial variation of the exposure.
Typically, temporal exposure variation involves sequentially capturing multiple images of the scene with different exposures and then combining the multiple captured images into a single high dynamic range image. The use of temporal exposure variation is disclosed, for example, in: Japanese Patent No. 08-331461 entitled “Driving Method for Solid-State Image Pickup Device” to Hamazaki, issued December 1996; Japanese Patent No. 07-254965 entitled “Electronic Image Pickup Device” to Saito, issued February 1995; U.S. Pat. No. 5,420,635 entitled “Video Camera, Imaging Method Using Video Camera, Method of Operating Video Camera, Image Processing Apparatus and Method, and Solid-State Electronic Imaging Device” to Konishi et al., issued May 1995; U.S. Pat. No. 5,455,621 entitled “Image Method for a Wide Dynamic Range and an Imaging Device for a Wide Dynamic Range” to Morimura, issued October 1993; Japanese Patent No. 06-141229 entitled “Image Picking Up and Synthesizing Method and Image Pickup Device with High Dynamic Range” to Morimura, issued May 1994; U.S. Pat. No. 5,801,773 entitled “Image Data Processing Apparatus for Processing Combined Image Signals in Order to Extend Dynamic Range” to Ikeda, issued September 1998; U.S. Pat. No. 5,638,118 entitled “Image Device with Diverse Storage Times Used in Image Composition” to Takahashi et al. issued June 1997; U.S. Pat. No. 5,309,243 entitled “Method and Apparatus for Extending the Dynamic Range of an Electronic Imaging System” to Tsai issued May 1994; Mann and Picard, “Being ‘Undigital’ with Digital Cameras: Extending Dynamic Range by Combining Differently Exposed Pictures,” Proceedings of IST's 48th Annual Conference, pp. 422–428, May 1995; Debevec and Malik, “Recording High Dynamic Range Radiance Maps for Photographs,” Proceedings of the ACM SIGGRAPH, 1997, pp. 369–378, August 1997; and Mitsunaga and Nayar, “Radiometric Self Calibration,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR 99), pp. 374–380, June 1999. However, known temporal exposure variation techniques all have the fundamental problem in that changes in the scene may take place during the sequential image capture at different exposures. Consequently, these techniques are useful only for static scenes where the radiance from the scene remain constant and where the position and orientation of the image sensor, and its geometric characteristics remain unchanged during sequential image capture at different exposures.
Another known technique for capturing multiple images of a scene for high dynamic range imaging uses multiple optically aligned image sensors and a beam splitter for providing incident light from the same view of the scene to the multiple image sensors. The image sensors having different sensitivities so that multiple images are simultaneously captured at different exposures. The multiple images captured by the image sensors are then combined to form a single high dynamic range image. Such techniques, for example, are described in: Japanese Patent No. 08-22341 entitled “Image Sensor” to Doi et al., issued August 1986; Japanese Patent No. 07-25495 entitled “Electronic Image Pickup Device” to Saito, issued February 1995; Japanese Patent No. 07-254966 entitled “Electronic Image Pickup Device” to Saito, issued October 1995; Japanese Patent No. 08-340486 entitled “Electronic Image Pickup Device” to Saito, issued December 1996; Japanese Patent No. 10-069011 entitled “Image Pickup Device” to Kimura, issued March 1998; and U.S. Pat. No. 5,801,773 entitled “Image Data Processing Apparatus for Processing Combined Image Signals in Order to Extend Dynamic Range” To Ikeda, issued September 1998. The multiple image sensor techniques have the drawback in that they require high precision optics and a plurality of image sensors so that it is difficult to build compact and inexpensive imaging systems using these techniques.
Known spatial exposure variation techniques for high dynamic range imaging typically use a special image sensor having more than one (typically two) light-sensing elements having different sensitivities for each pixel. The light-sensing elements for each pixel are simultaneously exposed and their respective output signals are combined to derive a brightness value for the pixel. Such techniques are described, for example, in: U.S. Pat. No. 5,789,737 entitled “High Dynamic Range Segmented Pixel Sensor Array” to Street et al., issued August 1998; U.S. Pat. No. 4,623,928 entitled “High Dynamic Range CCD Detector/Imager” to Handy, issued November 1986; U.S. Pat. No. 4,873,561 entitled “High Dynamic Range Charge Coupled Device” to Wen, issued October 1989; Japanese Patent No. 08-331461 entitled “Driving Method for Solid-State Image Pickup Device” to Hamazaki, issued December 1996; Japanese Patent No. 59-217358 entitled “Charge Coupling Image Pickup Device” to Murakoshi, issued December 1994; and U.S. Pat. No. 5,420,635 entitled “Video Camera Imaging Method Using Video Camera, Method of Operating Video Camera, Image Processing Apparatus and Method, and Solid-State Electronic Imaging Device” to Konishi et al., issued May 1995. However, known spatial exposure variation techniques for high dynamic range imaging have the drawback of providing lower spatial resolution (typically a factor of two lower in the vertical direction) than the resolution of ordinary image sensors having the same number of light-sensing elements.
Accordingly, there exists a need for a method and apparatus for capturing a relatively high dynamic range images using a relatively low dynamic range image sensor, which overcomes the problems of the prior art as discussed above.